Neural Information Processing Systems http://nips.cc/

In this section, we examine our theoretical results with controlled experiments via synthetic data. We do not have a complete explanation for such spikes. At first glance, overfitting could happen when the number of linear measurements is less than the size of the groundtruth matrix. Moreover, when the measurements satisfy RIP, Li et al. Soltanolkotabi [ 45 ] show that GD exactly recovers the ground truth. To our best knowledge, most existing generalization analysis for flat regularization are for two-layer models, e.g., Li et al.

We study implicit regularization when optimizing an underdetermined quadratic objective over a matrix X with gradient descent on a factorization of X. We conjecture and provide empirical and theoretical evidence that with small enough step sizes and initialization close enough to the origin, gradient descent on a full dimensional factorization converges to the minimum nuclear norm solution.

We study implicit regularization when optimizing an underdetermined quadratic objective over a matrix $X$ with gradient descent on a factorization of X. We conjecture and provide empirical and theoretical evidence that with small enough step sizes and initialization close enough to the origin, gradient descent on a full dimensional factorization converges to the minimum nuclear norm solution.

We study implicit regularization when optimizing an underdetermined quadratic objective over a matrix $X$ with gradient descent on a factorization of $X$. We conjecture and provide empirical and theoretical evidence that with small enough step sizes and initialization close enough to the origin, gradient descent on a full dimensional factorization converges to the minimum nuclear norm solution.